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Introduction 


U2 to DDR4 For Dummies, HP Special Edition, 
a book with everything you need to know about 
Double Data Rate 4 (DDR4) memory, the latest generation 
high-speed memory for the HP ProLiant Gen9 servers. 
You discover the memory population rules for HP’s Intel 
Xeon 4-way servers and the advantages of DDR4 over 
DDR3 memory. 


About This Gook 


This book comprises three parts: 


Y Part I: Getting to Know DDR4. You find out what’s 
new about DDR4 and how it compares to DDR3. 
(Hint: It’s faster. It uses less power.) You learn 
about the two main types of DDR4 Memory that 
are used in HP servers: 

R-DIMMs (Registered DIMMs) for price and 
performance 

LR-DIMMs (Load-Reduced DIMMs) for performance 
and maximum memory capacity. 

You’ll also learn how to identify DDR4 DIMMs and 
understand per-channel restrictions. 

Y Part II: Populating the Server. There are guide- 
lines for getting the best performance, and rules 
also exist about what works and what doesn’t. You 
ought to follow these guidelines, and you can’t 
break the rules. 
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Y Part III: Ten Common Questions about DDR4. 
This part finishes the book with some common 
questions. These are the “there are no stupid ques- 
tions” akin to the cruise ship passenger who asks, 
“Does the crew sleep on the ship?” 


Terms Used in This Book 


Before we get started, here are three acronyms (abbre- 
viations) you'll see a lot in this book that describe the 
types of DDR4 memory. 


DIMM stands for Dual Inline Memory Module, a module 
where the connectors, or gold fingers (not related to the 
1960s James Bond movie), at the front of the module are 
electrically separated from the ones at the back in order 
to utilize a wider data bus (whereas a SIMM, or Single 
Inline Memory Module, has the same amount of connec- 
tors at the front and the back of the module, but electri- 
cally shorted). 


R-DIMM, or Registered DIMM, uses a register (which acts 
as a buffer) on the address/command bus in order to 
put less of an electrical load on the system’s memory 
controller. It has nothing to do with filling out a registra- 
tion or warranty card. 

LR-DIMM, or Load-Reduced DIMM, is similar to an R-DIMM, 
but in addition to the register, it also uses a buffer on the 


data-bus to reduce the electrical load on the system’s 
memory controller. It’s faster. 


For more about the two different types of DIMM, see 
Part I. 


Icons Used in This Book 


This book uses icons to alert you to geeky, useful, and 
important factoids. 


ag) helps you make sense of some of the more 


gist, This icon alerts you to extra information that 
technical aspects in this book. 


This icon points out helpful and useful 


information. 

ome 

« This icon points out super-important 
information. 

Ry 


s This icon alerts you to information that may 
damage your system. Proceed with caution! 
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Part | 
Getting to Know DDR4 


In This Part 
Exploring DDR4 
Choosing between R-DIMMs and LR-DIMMs 
Looking at installation 
Understanding memory protection 
Identifying HP DIMMs 
Seeing DIMMs per channel restrictions 
Taking a look at HP part numbers 


A new generation of servers arrived in 2014 with 
new CPUs (Xeon v3). To maximize their perfor- 
mance, a new version of memory, called DDR4, was 
introduced. Spelled out, DDR4 SDRAM is short for 
double-data-rate fourth-generation, synchronous 
dynamic random access memory. (And you thought the 
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34 characters of “supercalifragilisticexpialidocious” 
was long!) DDR4 memory uses (as its predecessors 
DDR3, DDR2 and DDR1 already did) each of the two 
phases of the clock (rising and falling) to achieve the 
double data rate compared to what is now known as 
single data rate SDRAM memory. DDR4 is capable of 
supporting higher bandwidths than DDR3 already at 
introduction. 


oot s TUp,, 


43) Take a look at how DDR4 SDRAM DIMM is 


TEC, 


deconstructed: 


DDR (Double Data Rate Memory): It transfers 
twice the data per clock cycle versus non-DDR 


memory (which is now called single data rate 
or SDR). 


4 (Fourth Version of DDR): If you were trying to 
catch up with the Super Bowl, it would be DDR IV. 


¥ S Gynchronous): The memory accesses are syn- 
chronized with a memory clock. 


¥ D (Dynamic Memory): Unlike flash memory in 
your camera or a solid state drive, this memory 
needs to be electrically refreshed every few milli- 
seconds or, oops, no data. 


i RAM (Random Access Memory): Every bit can 
be accessed equally as fast, unlike a tape drive or 
file cabinet where you get to Aardvark before 
Zebra. 
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“ DIMM (Dual Inline Memory Module): Specs for 
64-bit processors needed two matched 32-bit 
single inline memory modules (SIMMs) to fill the 
64-bit data path; DIMMs have it all on one module. 


Why DDR&: Less Filling, 
Tastes Great 


You'll be happy to know that DDR4 continues the pace 
of energy efficiency. DDR3 started running at 1.5 volts; 
later, DDR3L (L for low voltage) ran even lower at 1.35V. 
DDR4 is now available starting at 1.2 volts, which in 
terms of power consumed (watts), is good for significant 
improvement over DDR3, which had improved con- 
sumption advantage over DDR2, as DDR2 had over 
DDRI1. 


Future DDR4 enhancements may drive power consump- 
tion even lower. Speed also matters. DDR4 helps make 
servers faster and more powerful. There are several 
reasons why you want DDR4 over DDR3 (beyond the 
fact that DDR3 won’t fit in DDR4 slots, and you’ll break 
something trying to prove otherwise): 


High performance: Performance of memory is 
usually characterized through latency (internal 
delays) and bandwidth (the rate data is read from 
or written to RAM). Although the overall latency 
hasn’t changed much over the last couple years 
(still around 13ns, or nanoseconds), the band- 
width certainly did. DDR4 enables memory to run 


(at introduction) at a 15 percent higher data rate 
than the maximum data rate of DDR3, which will 

increase up to 70 percent over the next couple of 
years. 


In Figure 1-1, you can see the DDR4 versus DDR3 
data rate (in MT/s). 





DRAM type Supported DATA Rates 








DDR3 800 | 1066 | 1333 | 1600 











1867 | ----- | a | | — 





























DDR4 —— | 1333 | 1600 | 1867 | 2133 | 2400 | 2667 | 3200 





Figure 1-1: DDR4 versus DDR3 data rates (in MT/s). 


Server support at DDR4 launch runs up to 2133MT/s 
(depending on configuration). Some refer to MT/s 
(mega-transfers per second) as MHz. For a 1066MHz 
clock the data transfer rate is 2133MT/s, hence 
double data rate. The two were the same when you 
moved one piece of data for each tick of the clock, 
but now you get double that, which is what this 
DDR thing is all about. Here, MHz means megahertz, 
not megatransfers. 


¥ Low power: Running at same speeds, the 1.2V 
DDR4 supply cuts power consumption by up to 
20 percent over DDR3 memory, which ran at 1.35V. 
Eagle-eyed math wizards will notice, that’s an 
11 percent drop in voltage, but let us explain: Other 
things are happening as well, and the power draw, 
expressed in watts, is down by about 20 percent. 
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High capacity: In DDR3, DIMMs from 2GB up to 
64GB were available, in DDR4, capacity starts at 
4GB up to 64GB and higher. This means, you’re 
able to run up to 3TB of memory in a DL380Gen9 
system with the release of 128GB DIMMs. 


Runs cooler: DDR4’s improved thermal character- 
istics allow each DIMM to run at lower tempera- 
tures then DDR3. DDR4 does its part to be neutral 
on climate change. 


Additional features worth mentioning include 


Write Data Cycle Cyclical Redundancy Check 
gives you better error detection capability and 
reliability but won’t be supported on server 
platforms because of ECC. 


Data Bus Inversion will reduce the power 
consumption and improve signal integrity. 


vy CA (Command/Address) Parity is a method to 
verify the integrity of command/address transfers 
(expected to be used eventually when U-DIMMs, 
or unregistered [unbuffered] DIMMs become 
available). 


When selecting DDR4 memory, options will include 


 R-DIMMs (higher performance, lower latency) 
modules: 4GB/8GB/16GB data rates: 
1333/1600/1867/2133 MT/s 


 LR-DIMMs (higher bandwidth, higher capacity) 
modules: 16GB/32GB/64GB data rates: 1333/1600/ 
1867/2133 MT/s 
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= DIMM speed depends on CPU capability, DIMM 
capability and number of DIMMs installed. 


¥ ECC (error-correcting code) 


Additional rules apply for HP servers. For details, see 
the sidebar “DDR4 for HP.” 








DDR4 for HP 
A few things make HP memory stand out: 
yw DDR4 memory was launched at 1.2 volts for Intel Xeon 
E5-2600 v3 series (2P servers) and Intel Xeon E7-4800 v3 
(4P servers). Memory sizes are 4GB/8GB/16GB/32GB and 
64GB. 


 OnHP servers, ProLiant Gen9 platforms with Intel pro- 
cessors support DDR4 R-DIMMs and LR-DIMMs but not 
mixed in the same unit. 

All HP servers support Advanced ECC for better data 
integrity. 

For HP servers, maximum capacity with 16GB R-DIMMs 
is as high as 384GB for 2P servers and now 1.5TB for 
4P servers. 


uw Memory modules carry a one-year warranty or assume 
the longer warranty of the server they're inside. 


HP DDR4 option kits contain a single DIMM module. 














1 
Choosing a DIMM Type 


Back in DDR3-land you had the choice among 3 different 
types of DIMMs (U-DIMMs, R-DIMMs, LR-DIMMs), the 
choices in DDR4 are narrowed down to two, R-DIMMs 
and LR-DIMMs. The main reason U-DIMMs are left out 
this time is that at higher DDR4 speeds you won't get 
any advantage over R-DIMMs anymore when it comes to 
performance and reliability. 
ft STUp, 


RY, Aes DDR4 memory is categorized by capacity, rank 
§\7 ) and organization: 


¥ Capacity is the total amount of storage on a single 
DIMM 


Rank is a group of DRAM chips that are accessed 
simultaneously (via the CS or chip select signal) 
to provide 72 bits of data (64 bits data + 8 Bits 
ECC) to the system. Used are 1R/2R/4R/8R. 


Organization determines the number of data bits 
coming out of a single DRAM. In DDR4 you will see 
x4 (pronounced “by four”) and x8. 


Figure 1-2 is an example of an HP 4GB and 8GB DDR4 
memory kit using the same DRAM density of 4 Gbit. 


TEC, 


As you probably notice, the HP memory kits for both 
DIMMs are labeled “PC4-2133P-R Kit.” This is slightly 
different from what you were probably used to in DDR3. 
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HP 4GB1Rx8 PC4-2133P-RKit 4 Gbit x 1 x 8= 4 GByte 
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HP 8GB2R x8 PC4-2133P-R Kit 4 Gbit x 2 x 8= 8 GByte 






























































Figure 1-2: An HP 4GB and an 8GB DDR4 memory kit. 


gone 


‘y) Here are the differences: 


i DRAM Data rate: This is the operating data rate for 
each bit on the DRAM. For example, DDR3-1600 has 
a data rate of 1600 MT/s (Mega Transfers per 
second). 

“ DDR3 DIMM Data Rate: The memory bus of a 
DIMM is 64 bit wide, which results in a DIMM Data 
Rate of 12.8 GB/s (1600 Mbit/s * 64 bit / 8 bits per 
Byte). The data rate for the whole DIMM is than 
labeled “PC3-12800.” Now you can easily derive all 
other speed-bins in DDR3, such as DDR3-1333 is 
PC3-10600. 
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“ DDR4 DIMM Data Rate: In DDR4 things become 
simpler, and the data rate for each DRAM is writ- 
ten “DDR4-2133.” For the label on the DIMM, it’s 
practically the same — “PC4-2133.” 


CAS latency: The DRAM response time from 
column address command to Ist data out on the 
memory bus. CAS latency is counted in clock 
cycles, so for example, a CL13 (pronounced “CAS 
latency of 13”) means the response of the DRAM is 
13 clock cycles. In Figure 1-3, you see how this 
works out by issuing a read command (RD) to the 
DRAM with a CL13. 


TO T. Tad TaS 


& DLINGINATAINATADIN 


Ta0 Tal 





cmp —{RD } : 
pata —{p0 pi (02 \p3 (D4 \(ps \ pe \ 07) 


4 cl=13 Pe 


Figure 1-3: An issued read command (RD) to the DRAM with 
a CL13. 





Of course you can imagine that the faster the clock runs 
by keeping the CL constant, the actual response time 
decreases. For the CAS latency, DDR3 used numbers 
such as PC3-12800R-13 where 13 was the CAS latency. In 
DDR4, you use letters starting at the letter P, which is 
CL15. From there you go on: Q = CL16, R = CL17, and so 
on. So, the label on a DDR4 tells you the following: 


PC4-2133P-R is a DDR4 DIMM with a data rate of 2133 MT/s 
per data line and a CAS latency of 15. What does the R 
stand for? Here we go... it’s finally the type of DIMM you 
can choose from, either R-DIMM (R) or LR-DIMM (L). 
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Registered DIMM (R-DIMM) 


Registered dual in-line memory module (R-DIMM) puts 
less electrical loading on the command/address signals 
due to the register on the DIMM between the memory 
controller and the DRAMs. The big difference is that 
LR-DIMM buffers Command/Address bus + data bus, 
R-DIMM only Command/address bus. 


The register captures the command/address signals 
from the memory controller and retransmits them to 
the DRAMs locally (provides greater reliability at a 
slight cost in performance of one clock cycle to affected 
signals). As a result, the system sees only one load per 
address line instead of 9, 18, or 36. Figure 1-4 shows an 
R-DIMM in a 2 SPC (slots per channel) system like the 
HP Proliant BL460 Gen9 populated with 1 DPC (DIMM 
per channel). 


DIMM 8 













1/23] 4|5]6|7|8|/9| 
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DRAMs 
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Memory Bus 
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VVVYYYYY! 











Figure 1-4: An R-DIMM in a 2 SPC (slots per channel) system. 


The data still flows in parallel as 72 bits (64 bit data + 
8 bit ECC) across the data portion of the memory bus, 
which becomes the limitation due to the electrical 
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loading when populating more DIMMs in a channel. In 
DDR3, R-DIMMs allowed for 1, 2, and 4 Ranks/DIMM sup- 
port, which caused some problems in a 3SPC system 

(3 times 4 Ranks per channel = 12 Ranks) because there 
was a restriction of maximum 8 Ranks per channel. 


In DDR4 you're limited to 1 and 2 Ranks/DIMM, so this 
restriction isn’t there anymore. R-DIMMs can be used in 
all HP Servers. Capacities are 4GB, 8GB, 16GB, and 
32GB (when available). In a 24-slot server, you could 
have as much as 384GB of DDR4 memory installed by 
using 16 GB R-DIMMs. To get higher memory capaci- 
ties, you need to look for another type of DIMM — the 
load-reduced DIMM. 


Load-reduced DIMM (LR-DIMM) 


LR-DIMM, or load-reduced DIMM, adds data buffers 
(DB) on the DIMM between the memory controller and 
DRAM in order to reduce electrical loading on the data 
signals of the memory bus. Figure 1-5 shows an 
LR-DIMM in a 2SPC system in a 1DPC configuration. 


DIMMO 
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Figure 1-5: An LR-DIMM in a 2SPC system in a 1DPC 
configuration. 
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The limitation of the R-DIMM (having a lot of electrical 
loading on the data signals when having more ranks or 
populating more DIMMs per channel) is now gone. This 
allows now for higher data rates on the memory chan- 
nel and also higher capacities on the DIMMs. 


Rn In DDR4, a new concept known as chip-select- 

4S) encoding is introduced. This allows the system 
to address multiple Ranks behind the LR-DIMM 
buffer by encoding the chip-select instead of 
using one chip-select per rank. 


With the advantage to run high capacity LR-DIMMs in 
the system at a higher bandwidth, the memory buffer 
ads additional latency to the data signals. LR-DIMMs 
come in different capacities. With the current avail- 
able DRAM density of 4 Gbit, you’ll see 16GB, 32GB, 
and 64GB LR-DIMMs. Later, with the release of 8 Gbit 
DRAMs, you see capacity going up to 128GB per DIMM. 
In a 24-slot server, you could have as much as 1.5 TB 
of DDR4 memory installed using 64 GB LR-DIMMs. 
With the introduction of the 128 GB LR-DIMM it will 
even go up to 3 TB. 


goes By the way, you have to choose one DIMM 
“| type. You can’t mix and match R-DIMM and 
LR-DIMM inside the same server. What hap- 
pens if you mix? The server won’t boot. 
Nothing gets damaged (except egos). 
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Take a look at the options in Table 1-1. 








Table 1-1 Choices in Memory Technology 
R-DIMMs LR-DIMMs 

Capacities 4GB, 8GB, 16GB, 16GB, 32GB, 64GB, 
(32GB) (128GB) 

Maximum DIMMs 3 dual rank 3 “any” rank 

per channel 

Address error Yes No 

detection 

Lower cost Yes No (due to add, 


buffer, and stack- 
ing technology) 











Ranks/DIMM 1,2,4 2,4,8 

support 

DRAM support x4, x8 x4, x8 

Maximum server 384 (24 slots) 1500 1500 (24 slots) 6000 
capacity (GB) (96 slots) (96 slots) 
Installation 


Installing DDR4 memory is physically easy, especially 
because the slots are color-coded on most HP servers. 
Memory slots are colored white and black and some- 
times blue; the white slots are where you install the 
first memory module on each channel. It does take a 
bit of advance work but not a lot. 
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Memory Protection 


ProLiant Gen9 servers offer several levels of memory pro- 
tection, similar to previous generations (see Table 1-2): 


ECC or error-correction code is the fundamental 
form of ProLiant memory protection. Parity, as in 
parity error (you very occasionally get parity 
errors on a desktop PC), tells you something went 
wrong, as in wrong and it can’t be made unwrong, 
so sorry. ECC is based on advanced mathematics, 
and there’s a fancy formula to tell you how much 
you need (log2(N)+1 where N is the number of 
data bits) or this simple explanation: To protect 
64 bits of data, you need 8 redundant bits. How 
good is it? ECC memory can detect and correct 
single-bit errors (the vast majority), can detect 
double-bit errors, and might detect (sorry, no 
guarantee) errors greater than double-bit. 


Advanced ECC (multiple-bit error detection) is 
ECC on steroids, except in this case steroids are 
good and won’t get you tossed out of the Olympics. 
Advanced ECC requires 16 redundant bits to pro- 
tect a 128-bit word. It’s able to detect and correct 
up to 4 data bits. With Advanced ECC, you can pro- 
tect against the loss of a single DRAM device. The 
data bus is divided into groups of 4 bits each, and 
any group can have all 4 bits in error so long as the 
other groups are okay. It’s able to detect 2-bit 
errors occurring in different groups. If 4-bit-wide 
DRAMs are used, then one DRAM represents one 
group on the bus. 
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¥ Mirrored memory protection is just like a mir- 
rored hard disk drive. You have twice as much as 
you need. Half is configured as system memory, 
and half is mirrored memory. Data is written to 
both places. Reads are from the system memory 
unless — you’ve probably guessed this — there’s 
an uncorrectable system error. 




















Table 1-2 Advanced Memory Protection 
Basic ECC HP Advanced ECC 
Technology Technology 

Device failure No Yes 

protection 

Industry-standard Yes Yes 

DIMMs 

Hot plug No No 

Failed DIMM — Offline Offline 

replacement 

Memory Offline Offline 

expansion 

Additional 0% 0% 


memory expense 
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Identifying HP DIMMs 


This section helps you read the fine print on product 
selector guides. HP certified memory will have these 
markings, from left to right: 


HP indicates it’s HP-approved memory being sold 
through an HP channel or direct from HP. 


 nGB indicates the capacity (n= 1, 2, 4, 8, 16, 32 or 
64 GB). 


eR indicates the number of ranks (e=1, 2 or 
4 Ranks). 


YY xf is the data width (f=4 or 8 bit). 

 PC4 is the memory type, PC4 being DDR4. 
 wwww is the module bandwidth (2133 MT/s). 
¥ Pis the CAS latency (P=15, Q=16, R=17). 

Ris the DIMM type (R=R-DIMM, L=LR-DIMM). 


Kit at the end indicates it’s an HP kit (and remem- 
ber that with DDR4 there’s one module per kit, 
not two). 


For example, 726718-B21 HP 8GB 1Rx4 PC4-2133P-R 
Kit indicates an HP DIMM with a 8GB capacity, single 
rank, a data width of 4, memory type of DDR4 with a 
data rate of 2133*8 MT/s, CAS latency of 15, and an 
HP kit. 
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DIMMs per Channel Restrictions 


A memory channel refers to DIMM slots tied to the 
same wires on the CPU. Multiple memory channels 
allow for faster operation, theoretically allowing 
memory operations to be up to four times as fast. Dual 
channel architecture with 64-bit systems provides a 
128-bit data path. Memory is installed in banks, and 
you have to follow a couple of rules to optimize perfor- 
mance. With the HP Intel E5-2600v3 and E7-4800v3 
series ProLiant Gen9 servers, there are four channels 
from each processor. 


Population rules must be followed or errors will occur. 
That is: 


#@ Same DIMM type; R-DIMM and LR-DIMM can’t be 
mixed in a system (meaning, you can’t populate 
one CPU with R-DIMMs, the other one with 
LR-DIMMs) 


v¥ Same size is recommended for performance; 
however, it’s possible to mix capacities 


vy Same speed 


Same technology (DDR4); DDR2 and DDR3 modules 
won't fit, physically, ina DDR4 socket due to the 
location of the notch of the DIMM and also of the 
outer edges being slightly beveled 


Channel rules and restrictions are covered in more 
detail in Part II. 
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HP Part Number Matters 
(Not Vendor Name) 


When buying memory through HP, remember that 
DIMMs don’t have to be from the same vendor. DIMMs 
are JEDEC standard. All DIMMs sourced through HP 
pass HP memory qualification tests. 

In brief: If the HP part number is the same, the DIMMs 
are compatible. 


Part ll 
Populating the Server 


In This Part 
Knowing rules to follow (and why) 
Populating in the proper order 


J n this part, we go over the nitty-gritty about popu- 
lating the memory slots. The rules may seem over- 
whelming at first. There is a lot of nitty and a lot of 
gritty. Some rules you have to follow to make sure the 
server runs. Others you want to follow to optimize per- 
formance. The rules vary depending on whether you’re 
installing R-DIMMs (registered DIMMs) or LR-DIMMs 
(load-reduced DIMMs) and how many memory slots 
your server has. Some are pretty obvious, at least in 
hindsight, such as: Don’t install memory connected to 
the second processor if you’ve only got one CPU 
installed in your server. 


What follows here is wisdom likely to survive the ages, 
or at least the next couple years, but always check 
online for the best and latest. 


For more information and the latest information on 
configuring your HP server with DDR4 memory, go to 
www.hp.com/go/memory. 
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Population Rules — HP DDR4 
ML, DL, and SL Servers 


These are the key rules to follow when populating 
memory slots on ProLiant Intel-based G9 servers. This 
is not the complete list. Still, it covers most HP ProLiant 
servers. Population rules for each of the HP Proliant 
servers can also be found in the HP QuickSpecs at 
http: //www.hp.com/go/memory. Also see Figure 2-1. 

















DIMM Type R-DIMM LR-DIMM 
DIMM Rank Single Rank Dual Rank Quad Rank 
DIMM Capacity 4GB 8GB 8GB 16GB 32GB 32GB 64GB 
DRAM Organization x8 x4 x8 x4 x4 x4 x4 
DIMM Native Speed (MT/s) 2133 2133 2133 2133 2133 2133 2133 








Maximum Capacity that can be achieved using these DIMMs 
16 slot servers (GB) 64 128 128 256 512 512 1024 
24 slot servers (GB) 96 192 192 384 768 768 1536 














Operating speed as a function of DIMM population (MT/s) 






































2 slots per channel, 1 DIMM loaded 2133 2133 2133 2133 2133 2133 2133 
2 slots per channel, 2 DIMMs loaded 2133 2133 2133 2133 2133 2133 2133 
3 slots per channel, 1 DIMM loaded 2133 2133 2133 2133 2133 2133 2133 
3 slots per channel, 2 DIMMs loaded 2133 1866 1866 1866 1866 2133 2133 
3 slots per channel, 3 DIMMs loaded 2133 1600 1600 1600 1600 1866 1866 
Notes 














LR-DIMMs will operate 1 speed bin faster than R-DIMMs in 3 slot per channel servers. 





Figure 2-1: Memory options for HP DDR4. 


For the Proliant Intel-based G9 servers, the chipset sup- 
ports a maximum of 4 DDR4 channels per CPU socket. 
Following the Intel documentation naming convention, 
these channels are numbered from 0 through 3. 


For servers with 24 memory slots, there are three DIMM 
slots for each memory channel; 24 total slots (two CPU 
servers). For servers with 16 memory slots, there are 
two DIMM slots for each memory channel; 16 total slots 
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(two CPU servers). For servers with 12 memory slots, 
there are three DIMMs slots for each memory channel; 
12 total slots (one CPU server). 


The following bullets describe the memory subsystem 
and the population rules. 


Memory Channels 1 and 3 comprise the three 
DIMMs closest to the processor. Memory Channel 
0 and 2 comprise the three DIMMs farthest from 
the processor. 


Populate the white DIMM slots first. 
Don’t mix LR-DIMMs with R-DIMMs. 


¥¥ Don’t install DIMMs if there’s no corresponding 
processor installed. If there’s only one CPU ina 
two-CPU system, only half the DIMM slots are 
available. This one is so obvious when you’re 
reading advice, less so when you have all those 
available memory slots staring you in the face. 


You can have up to twelve ranks installed per 
channel. 


You can install three quad-rank DIMMs per 
channel. 


~ To maximize performance 


e Balance the memory capacity among installed 
processors. 


e Use the same type and same capacity DIMMs 
to optimize the interleaving scheme. 


¢ Populate all the memory channels to make use 
of all the bandwidth available in the memory 
subsystem. 
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Populate DIMMs from heaviest load (quad-rank) 
to lightest load (single-rank) within a channel. The 
heaviest load (DIMM with most ranks) within a 
channel goes farthest from the chipset. 


Memory mirroring is the mechanism by which the 
memory is mirrored, creating two identical copies of the 
data in the main and target copy. If the DIMMs were pol- 
ished chrome and the mirrored channels faced each 
other, you could almost see to infinity. When mirroring 
is enabled, the memory image on the memory channel 0 
is exactly duplicated on the memory channel 1 and simi- 
larly channel 2 is duplicated on channel 3. The DIMM 
configuration on the mirrored channels should be iden- 
tical. Note that only half of the memory capacity of both 
memory channels is available. The scope of the mirror- 
ing is limited to within a processor socket. 


Mixing DIMM speeds is allowed, but DIMMs will run at 
the slower of the two speed ratings. 


DIMM Socket Color Coding 


In addition to the DIMM numbering scheme, a DDR 
channel will have unique colors per DIMM socket. The 
first DIMM will be populated in the white socket, and 
the second DIMM will be populated in the black socket. 
If a third DIMM is present on that channel, that DIMM 
will be populated in the blue socket. 


a) In dim light, black looks a lot like blue. Your 
work area should be well lit. If necessary, sup- 
plement room light with a flashlight or your 
phone’s flashlight utility. 


Figure 2-2 shows the color and loading scheme for the 
DIMMs in HP’s DDR4 systems. 
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Figure 2-2: DIMM Socket Color Scheme. 


Populating Alphabetically 


HP recommends populating the DIMMs in alphabetical 
order to ensure best DIMM distribution in the system 
and to maximize server performance. This method 
ensures that the DIMMs are spread between the pro- 
cessors installed in the system and the available DDR 
channels, maximizing the use of memory bandwidth 
resources. Populating memory using this rule ensures 
that the available memory bandwidth resources are uti- 
lized and the memory interleaving is optimized. 


For a quick guide on how to populate your slots for a 
16 DIMM slot server, check Figure 2-3. Install DIMMs in 
alphabetical order. Look at the Population Order 
column. You'd install into the A slot first, installing into 
Slot 2, then place B into slot 4 and so on down the 
alphabet. The rules are the same for CPU1 and CPU2. 
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[Siot# [Population Order | 
Channel 
cna? LA) | Cd 
[aes | | J 
[jee | | | 
deo) | | _J 
cama Loses) | |_| 
dem) | | 
camera Low) | |_| 
dee) | | 1 
| ___|@PUt- Present __|cPu2—Present___| 
came La) | ae) _ 
Pale) | _alevro) _] 
[eo | eu) _| 
Pde) | eco) _] 
camaa Lo see) | slower 
Pao) | aoa) _| 
camee Lor) | ow) | 
anc) | co) _| 


Figure 2-3: 16 DIMM Slots CPU1 and CPU2. 





For a quick guide on how to populate your slots for a 
24 DIMM slot server, check Figure 2-4. Install DIMMs in 
alphabetical order. Look at the Population Order 
column. You'd install into the A slot first, installing into 
slot 3, then place B into slot 6, and so on down the 
alphabet. The rules are the same for CPU1 and CPU2. 


Much more information is available via the HP DDR4 online 
configuration tool: http: //www.hp.com/go/memory. 


CPU1 - Present CPU2- Not Present 
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Figure 2-4: 24 DIMM Slots for CPU1 and CPU2. 
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Part lil 


Ten Common Questions 
about DDR4 


In This Part 
Figuring out what DDR4 is 
Deciding between R-DIMM and LR-DIMM 
Knowing what mistakes to avoid 


Te Part of Tens is a For Dummies tradition and has 
outlived Letterman’s Top Ten lists. In this part, we 
answer some of your most pressing questions. All the 
wisdom you need, in a nutshell. 


What Is DDR4 Memory? 


Double-data-rate (DDR) is a form of SDRAM (synchro- 
nous dynamic random access memory), dynamic mean- 
ing that the memory is volatile and it forgets what’s 
stored when you pull the power plug, unlike ROM or 
Flash or other forms of non-volatile memory. The 
double part means data is transmitted twice per clock 
cycle. 
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What Does the 4 Stand For? 
And What's the Big Deal? 


This is the fourth iteration of double data rate memory, 
and it is faster, lower power and higher capacity than 
the previous generations. Before there was DDR3 there 
was DDR2 which was preceded by DDR, now in hind- 
sight called DDR1. The main benefits of DDR4 memory 
are its lower power and higher data transfer rates — it 
provides better performance at a lower cost of 
ownership. 


How Many Kinds of DDR4 
Memory Are There? 


For server use there are two main types of DDR4 
DIMMs: 


¥ Registered DIMMs (R-DIMMs) come in densities of 
AGB to 32GB, operate at speeds of up to 2133MT/s 
and are the lower cost and lower power DDR4 
DIMMs. 


 Load-reduced DIMMS (LR-DIMMs) come in densi- 
ties of 32 GB and 64GB and are the performance 
DIMMs operating one speed bin higher than the 
R-DIMMs in 2 and 3 DIMM per channel 
configurations. 


The larger capacity LR-DIMMs are also preferred for 
systems that run OLTP or in memory database applica- 
tions, providing a place to store large amounts of data 
and improving the system overall performance by 
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limiting the amount of time data has to be fetched from 
the significantly slower HDD (hard disk drive) or SSD 
(solid state drive). 


Other DDR4 DIMM types which are not frequently used 
in servers are 
~ U-DIMMs, unbuffered DIMMs (low capacity, low 
speed, low cost) 


¥ So-DIMMs, small outline DIMMS, mainly used in 
workstations and PCs 


Do I Want R-DIMM, LR-DIMM 
for Price? For Performance? 


LR-DIMMs are your choice for capacity (up to 1.5TB ina 
qualified 24-slot Intel server) and performance. R-DIMMs 
are your choice for lower power and lower cost. 


LR-DIMMs are always built with x4 DRAM devices. The 
majority of the R-DIMMS are also built with x4 DRAM 
devices. Choose x4 DIMMs for best reliability capability 
support such as Advanced ECC or DDDC+1, depending 
on system and chipset support. This will minimize the 
system downtime by protecting you from memory errors. 


Is LR-DIMM or R-DIMM 


Greener? 


The good news is that DDR4 DIMMs use significantly less 
power than DDR3 DIMMs of the same capacity and 
speed. Among DDR4 types, the R-DIMMs are built with 
fewer DRAM devices and will consume less power than 
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the LR-DIMMs, so the overall memory subsystem power 
will be lowest when using R-DIMMs. For bandwidth inten- 
sive benchmarks for which the overall performance isn’t 
a function of the total memory capacity, the R-DIMM will 
provide the best performance per watt. For benchmarks 
like TPC-C where the overall performance is a function of 
the total memory installed in the server, the LR-DIMMs 
will provide best TPC-C. 


Which Works in Which HP 
ProLiant G9 Server? 


All R-DIMMs and LR-DIMMs covered in this book work 
with the Proliant G9 servers. But choose one; you can’t 
mix ’n’ match. For product offerings, visit www.hp.com. 


Is There DDR4 Memory 
for HP AMD Servers? 


HP ProLiant servers with AMD processors will offer 
systems using DDR4 memory. Stay tuned. 


How Else Could I Mess Up? 


You can’t mix and match LR-DIMMs and R-DIMMs on the 
same machine. And you can’t use older technology 
memory (DDR2 or DDR3) in ProLiant G9 platform servers: 
Older technology DIMMs don’t fit because they are keyed 
to be prevented from being inserted in a DDR4 slot. 
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Does a DDR3 DIMM Fit 
in a DDRG Server? 


NING, 
RY 


No. It won’t fit (different pin configuration, 
different notch location). If you push it in, 
you'd destroy the memory module or socket. 
If you make it fit without damage, the contacts 
wouldn’t line up and you'd create a short. Will 
left-over DDR3 work in a DDR4 server? No, and 
for the same reasons. It. Will. Not. Go. 


Will DDR4 RAM Speed Up 
My Old DDR3 Server? 


ING, 


RY 


The DDR4 DIMM has a completely different 
pinout and key location compared to a DDR3 
DIMM. For this reason, it will not fit in a DDR3 
connector, and if forced it will damage the 
server either by creating shorts or by causing 
mechanical damage. 


To take advantage of the DDR4 lower power and better 
performance, you need to purchase a DDR4 server. 


What about DDRG PCs? 


Let me guess: You’ve got a gamer in the family. The 
short answer is yes; there will be a new generation of 
higher-end CPUs, motherboards, and PCs that will sup- 
port DDR4 memory for workstations and, of course, 
gaming fanatics. There will be some performance 
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improvement. As with servers, you can’t just drop 
DDR4 memory modules into an existing PC running 
DDR3 memory. 


Is There a DDR5? 


You had to ask! JEDEC (Joint Electron Device Engineering 
Council), which is creating the global open standards for 
the microelectronics industry, is working on the defini- 
tion of the memory devices to be used in the future. 
Since DDR¢4 is just coming to market, the next generation 
of memory standards is far on the horizon. Our crystal 
ball can’t tell us if we are looking at DDR5 or another 
future memory technology yet — stay tuned. 


Re If you’re one for trend lines (past performance 

4S) is no predictor of future performance, as the 
brokerage ads say), each iteration of DDR 
memory was 4 to 5 years apart, so a fifth point 
on the plot would be toward the end of this 
decade. That’s a long time from now. 
Computer years are like dog years. 


Why Should 1 Buy DDR4 
Memory from HP? 


It’s possible to find DDR4 RAM from sources other than 
HP. Being the largest buyer of DRAM on the planet, HP 
has the ability to choose and qualify only the highest- 
quality DRAM. The memory HP rejects doesn’t get 
scrapped but, instead, gets resold to other OEMs and 
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third party module manufacturers. HP Memory is tested, 
tuned, and optimized on HP ProLiant-branded servers, 
not just test equipment. This ensures utmost reliability 
and reduces memory errors related to board level signal 
integrity issues. HP stands behind the memory it recom- 
mends and sells. HP DDR4 memory carries the warranty 
of the server, up to three years. If there’s a performance 
issue, you don’t suffer from multiple-vendor finger point- 
ing. Memory is just one part of the hardware cost, and 
hardware cost is just one part of the total cost of running 
the server over its useful lifetime. 


*) For more information on HP Qualified Server 
Memory or other HP Qualified Options visit 
www.hp.com/go/hpqo. 
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